Skip to content

[8.19] (backport #15427) Override fleet.ssl.certificate_authorities from env vars in container mode#15468

Open
mergify[bot] wants to merge 1 commit into
8.19from
mergify/bp/8.19/pr-15427
Open

[8.19] (backport #15427) Override fleet.ssl.certificate_authorities from env vars in container mode#15468
mergify[bot] wants to merge 1 commit into
8.19from
mergify/bp/8.19/pr-15427

Conversation

@mergify

@mergify mergify Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

What does this PR do?

When in container mode, override fleet.ssl.certificate_authorities from env vars FLEET_CA, KIBANA_CA, and ELASTICSEARCH_CA instead of using saved fleet.enc values.

Follows the same first-set-wins fallback chain used in setup_config.go.

Why is it important?

Without this override, an agent running in container mode will crash during startup if fleet.enc still contains a fleet.ssl.certificate_authorities path that no longer exists on disk.

Checklist

  • I have read and understood the pull request guidelines of this project.
  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • [ ] I have made corresponding changes to the documentation
  • [ ] I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in ./changelog/fragments using the changelog tool
  • I have added an integration test or an E2E test

Disruptive User Impact

N/A

How to test this PR locally

Unit test: TestContainerEnvOverridesFleetTLSPaths
Integration test: TestContainerCMDTLSCertOverride
Manually: Using kind, deploy ECK cluster. Enroll an agent with FLEET_SERVER_CLIENT_AUTH=required and FLEET_CA pointing to a mounted CA bundle. Unmount the CA bundle, unset env vars, and verify that agent restarts healthy.

Related issues

Closes #15408
Relates #14408

This is an automatic backport of pull request #15427 done by Mergify.

@lorienhu @mergify
Override fleet.ssl.certificate_authorities from env vars in container…
a4ba132 
… mode (#15427)

Reads fleet.ssl.certificate_authorities from the first-set of FLEET_CA, KIBANA_CA, ELASTICSEARCH_CA over potentially stale fleet.enc values.

(cherry picked from commit 3ca03cf)
@mergify mergify Bot added the backport label Jul 11, 2026
@mergify mergify Bot requested a review from a team as a code owner July 11, 2026 12:46
@mergify mergify Bot requested review from lorienhu and ycombinator and removed request for a team July 11, 2026 12:46
@github-actions github-actions Bot added the bug Something isn't working label Jul 11, 2026
@infra-vault-gh-plugin-prod

infra-vault-gh-plugin-prod Bot commented Jul 11, 2026 •
edited
Loading

Copy link
Copy Markdown

💔 Build Failed

Failed CI Steps

History

cc @lorienhu

@github-actions

github-actions Bot commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

TL;DR

The FIPS-only unit-test job fails deterministically in the new TestContainerEnvOverridesFleetCA test, not in the container CA override implementation. The test creates and writes an encrypted fleet.enc; vault encryption uses an arbitrary GCM IV, which Go rejects under GODEBUG=fips140=only.

Remediation

  • Add fipsutils.SkipIfFIPSOnly(t, "vault does not use NewGCMWithRandomNonce") at the start of TestContainerEnvOverridesFleetCA, following the existing pattern in internal/pkg/crypto/io_aesgcm_replace_test.go:303, or refactor the fixture to avoid the non-FIPS vault/storage writer.
  • Keep the non-FIPS unit/integration coverage for the override behavior and rerun GODEBUG="fips140=only,tlsmlkem=0" .buildkite/scripts/steps/unit-tests.sh.
Investigation details

Root Cause

internal/pkg/agent/cmd/container_test.go:1390 calls secret.CreateAgentSecret, and :1392-1393 creates an encrypted disk store used by the test cases. The first case then calls store.Save, which reaches vault encryption. The failing test is TestContainerEnvOverridesFleetCA at container_test.go:1315 in the Buildkite checkout. The production changes in internal/pkg/agent/cmd/container.go are not on the failing stack; the failure occurs during test setup.

The repository’s internal/pkg/testutils/fipsutils.SkipIfFIPSOnly explicitly skips tests that require legacy NewGCM behavior, and the existing crypto compatibility test uses this guard.

Evidence

  • Build: https://buildkite.com/elastic/elastic-agent/builds/42813
  • Job/step: Unit tests - fips140=only Ubuntu 22.04
  • Key log excerpt: vault Set: could not encrypt key: crypto/cipher: use of GCM with arbitrary IVs is not allowed in FIPS 140-only mode, use NewGCMWithRandomNonce
  • Result: DONE 3306 tests, 67 skipped, 1 failure

Verification

  • The failure is reproduced in the provided Buildkite log at internal/pkg/agent/cmd/container_test.go:1315.
  • No matching flaky-test issue was found, and the PR has no prior detective comment for this root cause.

What is this? | From workflow: PR Buildkite Detective

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ycombinator ycombinator Awaiting requested review from ycombinator ycombinator is a code owner automatically assigned from elastic/elastic-agent-control-plane
@lorienhu lorienhu Awaiting requested review from lorienhu lorienhu is a code owner automatically assigned from elastic/elastic-agent-control-plane

At least 1 approving review is required to merge this pull request.

Assignees

@lorienhu lorienhu

Labels

backport bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@lorienhu